{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "view-in-github"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/AI4Finance-LLC/ElegantRL/blob/master/eRL_demo_StockTrading.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c1gUG3OCJ5GS"
      },
      "source": [
        "# **Stock Trading Application in ElegantRL**\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FGXyBBvL0dR2"
      },
      "source": [
        "# **Part 1: Problem Formulation**\n",
        "Formally, we model stock trading as a Markov Decision Process (MDP), and formulate the trading objective as maximization of expected return:\n",
        "\n",
        "\n",
        "\n",
        "*   **State s = [b, p, h]**: a vector that includes the remaining balance b, stock prices p, and stock shares h. p and h are vectors with D dimension, where D denotes the number of stocks. \n",
        "*   **Action a**: a vector of actions over D stocks. The allowed actions on each stock include selling, buying, or holding, which result in decreasing, increasing, or no change of the stock shares in h, respectively.\n",
        "*   **Reward r(s, a, s’)**: The asset value change of taking action a at state s and arriving at new state s’.\n",
        "*   **Policy π(s)**: The trading strategy at state s, which is a probability distribution of actions at state s.\n",
        "*   **Q-function Q(s, a)**: the expected return (reward) of taking action a at state s following policy π.\n",
        "*   **State-transition**: After taking the actions a, the number of shares h is modified, as shown in Fig 3, and the new portfolio is the summation of the balance and the total value of the stocks."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jESlu_WtD_JS"
      },
      "source": [
        "# **Part 2: Stock Trading Environment Design**\n",
        "\n",
        "**State Space and Action Space**\n",
        "\n",
        "\n",
        "*   **State Space**: We use a 181-dimensional vector consists of seven parts of information to represent the state space of multiple stocks trading environment: [b, p, h, M, R, C, X], where b is the balance, p is the stock prices, h is the number of shares, M is the Moving Average Convergence Divergence (MACD), R is the Relative Strength Index (RSI), C is the Commodity Channel Index (CCI), and X is the Average Directional Index (ADX).\n",
        "*   **Action Space**: As a recap, we have three types of actions: selling, buying, and holding for a single stock. We use the negative value for selling, positive value for buying, and zero for holding. In this case, the action space is defined as {-k, …, -1, 0, 1, …, k}, where k is the maximum share to buy or sell in each transaction.\n",
        "\n",
        "\n",
        "**Easy-to-customize Features**\n",
        "\n",
        "\n",
        "*   **initial_capital**: the initial capital that the user wants to invest.\n",
        "*   **tickers**: the stocks that the user wants to trade with.\n",
        "*   **initial_stocks**: the initial amount of each stock and the default could be zero.\n",
        "*   **buy_cost_pct, sell_cost_pct**: the transaction fee of each buying or selling transaction.\n",
        "*   **max_stock**: the user is able to define the maximum number of stocks that are allowed to trade per transaction. Users can also set the maximum percentage of capitals to invest in each stock.\n",
        "*   **tech_indicator_list**: the list of financial indicators that are taken into account, which is used to define a state.\n",
        "*   **start_date, start_eval_date, end_eval_date**: the training and backtesting time intervals. Thee time dates (or timestamps) are used, once the training period is specified, the rest is backtesting.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DbamGVHC3AeW"
      },
      "source": [
        "# **Part 3: Install ElegantRL and related packages**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "U35bhkUqOqbS",
        "outputId": "e67021c0-4daf-408c-f6c2-e34b442ad45a"
      },
      "outputs": [],
      "source": [
        "!pip install git+https://github.com/AI4Finance-LLC/ElegantRL.git\n",
        "!pip install yfinance stockstats"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UVdmpnK_3Zcn"
      },
      "source": [
        "# **Part 4: Import Packages**\n",
        "\n",
        "\n",
        "*   **ElegantRL**\n",
        "*   **yfinance**: yfinance aims to solve this problem by offering a reliable, threaded, and Pythonic way to download historical market data from Yahoo! finance.\n",
        "*   **StockDataFrame**: stockstats inherits and extends pandas.DataFrame to support Stock Statistics and Stock Indicators.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1VM1xKujoz-6"
      },
      "outputs": [],
      "source": [
        "from elegantrl.run import *\n",
        "from elegantrl.agent import AgentPPO, AgentDDPG\n",
        "from elegantrl.envs.FinRL.StockTrading import StockTradingEnv, check_stock_trading_env\n",
        "import yfinance as yf\n",
        "from stockstats import StockDataFrame as Sdf"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3n8zcgcn14uq"
      },
      "source": [
        "# **Part 5: Specify Agent and Environment**\n",
        "\n",
        "*   **args.agent**: firstly chooses one DRL algorithm to use from agent.py. In this application, we prefer to choose DDPG and PPO agent.\n",
        "*   **args.env**: creates the environment, and the user can either customize own environment or preprocess environments from OpenAI Gym and PyBullet Gym from env.py. In this application, we create the self-designed stock trading environment.\n",
        "\n",
        "\n",
        "> Before finishing initialization of **args**, please see Arguments() in run.py for more details about adjustable hyper-parameters.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "E03f6cTeajK4",
        "outputId": "2e47cb16-baa0-487d-ca71-ac6c2d249e6b"
      },
      "outputs": [],
      "source": [
        "# Environment\n",
        "tickers = [\n",
        "  'AAPL', 'ADBE', 'ADI', 'ADP', 'ADSK', 'ALGN', 'ALXN', 'AMAT', 'AMD', 'AMGN',\n",
        "  'AMZN', 'ASML', 'ATVI', 'BIIB', 'BKNG', 'BMRN', 'CDNS', 'CERN', 'CHKP', 'CMCSA',\n",
        "  'COST', 'CSCO', 'CSX', 'CTAS', 'CTSH', 'CTXS', 'DLTR', 'EA', 'EBAY', 'FAST',\n",
        "  'FISV', 'GILD', 'HAS', 'HSIC', 'IDXX', 'ILMN', 'INCY', 'INTC', 'INTU', 'ISRG',\n",
        "  'JBHT', 'KLAC', 'LRCX', 'MAR', 'MCHP', 'MDLZ', 'MNST', 'MSFT', 'MU', 'MXIM',\n",
        "  'NLOK', 'NTAP', 'NTES', 'NVDA', 'ORLY', 'PAYX', 'PCAR', 'PEP', 'QCOM', 'REGN',\n",
        "  'ROST', 'SBUX', 'SIRI', 'SNPS', 'SWKS', 'TTWO', 'TXN', 'VRSN', 'VRTX', 'WBA',\n",
        "  'WDC', 'WLTW', 'XEL', 'XLNX']  # finrl.config.NAS_74_TICKER\n",
        "\n",
        "tech_indicator_list = [\n",
        "  'macd', 'boll_ub', 'boll_lb', 'rsi_30', 'cci_30', 'dx_30',\n",
        "  'close_30_sma', 'close_60_sma']  # finrl.config.TECHNICAL_INDICATORS_LIST\n",
        "\n",
        "gamma = 0.99\n",
        "max_stock = 1e2\n",
        "initial_capital = 1e6\n",
        "initial_stocks = np.zeros(len(tickers), dtype=np.float32)\n",
        "buy_cost_pct = 1e-3\n",
        "sell_cost_pct = 1e-3\n",
        "start_date = '2008-03-19'\n",
        "start_eval_date = '2016-01-01'\n",
        "end_eval_date = '2021-01-01'\n",
        "\n",
        "\n",
        "# Agent\n",
        "env = StockTradingEnv('./', gamma, max_stock, initial_capital, buy_cost_pct, \n",
        "                           sell_cost_pct, start_date, start_eval_date, \n",
        "                           end_eval_date, tickers, tech_indicator_list, \n",
        "                           initial_stocks, if_eval=False)\n",
        "agent = AgentPPO()  # AgentSAC(), AgentTD3(), AgentDDPG()\n",
        "args = Arguments(agent=agent, env=env)\n",
        "args.agent.if_use_gae = True\n",
        "args.agent.lambda_entropy = 0.04\n",
        "\n",
        "args.env_eval = StockTradingEnv('./', gamma, max_stock, initial_capital, buy_cost_pct, \n",
        "                           sell_cost_pct, start_date, start_eval_date, \n",
        "                           end_eval_date, tickers, tech_indicator_list, \n",
        "                           initial_stocks, if_eval=True)\n",
        "\n",
        "args.env.target_reward = 3\n",
        "args.env_eval.target_reward = 3\n",
        "\n",
        "# Hyperparameters\n",
        "args.gamma = gamma\n",
        "args.break_step = int(2e5)\n",
        "args.net_dim = 2 ** 9\n",
        "args.max_step = args.env.max_step\n",
        "args.max_memo = args.max_step * 4\n",
        "args.batch_size = 2 ** 10\n",
        "args.repeat_times = 2 ** 3\n",
        "args.eval_gap = 2 ** 4\n",
        "args.eval_times1 = 2 ** 3\n",
        "args.eval_times2 = 2 ** 5\n",
        "args.if_allow_break = False\n",
        "args.rollout_num = 2 # the number of rollout workers (larger is not always faster)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z1j5kLHF2dhJ"
      },
      "source": [
        "# **Part 6: Train and Evaluate the Agent**\n",
        "\n",
        "> The training and evaluating processes are all finished inside function **train_and_evaluate_mp()**, and the only parameter for it is **args**. It includes the fundamental objects in DRL:\n",
        "\n",
        "*   agent,\n",
        "*   environment.\n",
        "\n",
        "> And it also includes the parameters for training-control:\n",
        "\n",
        "*   batch_size,\n",
        "*   target_step,\n",
        "*   reward_scale,\n",
        "*   gamma, etc.\n",
        "\n",
        "> The parameters for evaluation-control:\n",
        "\n",
        "*   break_step,\n",
        "*   random_seed, etc.\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KGOPSD6da23k",
        "outputId": "2fdbfc4b-80a2-4659-ace5-b3bde0f36d8e"
      },
      "outputs": [],
      "source": [
        "train_and_evaluate_mp(args) # the training process will terminate once it reaches the target reward."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JPXOxLSqh5cP"
      },
      "source": [
        "Understanding the above results::\n",
        "*   **Step**: the total training steps.\n",
        "*  **MaxR**: the maximum reward.\n",
        "*   **avgR**: the average of the rewards.\n",
        "*   **stdR**: the standard deviation of the rewards.\n",
        "*   **objA**: the objective function value of Actor Network (Policy Network).\n",
        "*   **objC**: the objective function value (Q-value)  of Critic Network (Value Network)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v6jvgYPnHMpf"
      },
      "source": [
        "# **Part 7: Backtest and Draw the Graph**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "3jCdezgzQUhh",
        "outputId": "a1ea5861-1ce9-4d44-c94d-3c236eb48428"
      },
      "outputs": [],
      "source": [
        "agent = AgentPPO()\n",
        "env = StockTradingEnv(cwd='./', if_eval=True)\n",
        "args = Arguments(agent=agent, env=env)\n",
        "args.if_remove = False\n",
        "args.cwd = './AgentPPO/StockTradingEnv-v1_0'\n",
        "args.init_before_training()\n",
        "\n",
        "env.draw_cumulative_return(args, torch)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "include_colab_link": true,
      "name": "eRL_demo_StockTrading.ipynb",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.5"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
